Fix Int overflow on 32-bit platforms (wasm32, armv7) - #88
Conversation
|
Yes, valid point. The library was built primarily targeting 64-bit platforms but after the cross-platform effort invariably there are still 32-bit targets in this day and age. WASM64 is in the works but likely not viable any time soon. Forgive my possible naïveté, but |
That would actually be fantastic if you would like to. Preferably branch off main and punt it over as a new PR even if the actual tests fail. I've only recently added Android and now WASM build jobs to repository CI pipelines, but just haven't had time to look into how to get actual unit tests happening on CI. |
83bd6d1 to
597a0c4
Compare
|
Not naïve at all — you're right, and it's the better fix. I've force-pushed it onto this branch (the saturating version is gone; shout if you'd rather have had it as a separate PR and I'll restore). No platform-conditional logic anywhere. The internal domain widens cleanly because the
Deliberately not widened: One narrowing remains, commented at the site: Also correcting something I wrote in the original description: I said the product exceeds Verified: full suite 506 tests in 55 suites on macOS, and 551 tests in 128 suites on wasm32 under wasmtime — the latter by pinning this branch into my own project, which was the only way I could actually execute your library's code on wasm. Which leads into the CI follow-up you asked for; PR coming, and it explains why running your suite there is not yet a two-line job. Unlike the saturating version, this makes |
| let outFrames = (subFrames - outSubFrames) / base.rawValue | ||
| static func subFramesToFrames(_ subFrames: Int64, base: SubFramesBase) -> (frames: Int, subFrames: Int) { | ||
| // The COUNT needs 64 bits; the resulting frames/subFrames do not — | ||
| // max total frames is ~1.04e9 even at 120 fps over 100 days. |
There was a problem hiding this comment.
This holds true at present time, but we can't assume 120fps will remain the highest frame rate provided by the library, nor should we assume that 100 days the greatest maximum upper bound that will be implemented. At 100 days, 120fps occupies 30 bits + 1 for the sign bit. 240fps occupies 31 bits + the sign bit. 480fps overflows a 32-bit signed Int.
Unit tests will invariably trip at any future point if greater frame rates or upper bounds are supported of course but it may be worth considering at this stage.
My feeling that there should be consistency with consumed and emitted types concerning total frame counts and total subframe counts across the library's public API surface. Inconsistency may be confusing to the consumer if a frame count is typed as You may have found them, but for cleanness and conciseness, deprecations all belong in a respective target's
In keeping with the concern for consistency, |
|
Agreed on consistency — a frame count typed One finding worth folding into the scope before anyone starts, which came out of the WASM CI work in #89: there is a third total-count domain in the same family — audio samples. That makes the job meaningfully larger than this PR, which is why I want to ask rather than assume: would you prefer to merge this one as approved and take the consistency work as a follow-up PR, or hold this one and do it all together? I am happy either way and will do the work regardless — it is your API and your call on how to stage it. My only reason for raising it is that this PR is already approved and fixes a hard trap on 32-bit, so there may be value in it landing on its own rather than waiting behind a larger refactor. If you would rather have one coherent change, say so and I will fold it all in here, Unrelated, in case it is useful: |
Yes, good point.
Well aware. GitHub CI has been very unreliable and often the runners and Actions backend cause random test failures and cancellations. It's a constant game of plugging leaks in the dam because the runners are a moving target and their composition never stays static for long. |
Int overflow on 32-bit platforms (wasm32, watchOS)
I think we can add it to this PR, as it's closely related in scope. The commit history is enough to allow in-situ rollbacks if needed. |
FYI: I was right (#90). The runners are broken. There's nothing wrong with the package or the CI job itself. GitHub wastes so much of my time chasing false positives it's beyond belief. |
|
Consistency work pushed — frames, subframes, samples and The rule I applied throughout: totals are Widened — Left as A nice side effect: widening Two things needing your call1. 2. No
If you want migration cover for those, it needs differently-named accessors — something like Worth noting the tests caught real 32-bit problems on the way through, not just type churn: several sample-count literals in |
|
Thanks very much.
Not strictly the case. func foo() -> Int { 0 }
@_disfavoredOverload
func foo() -> Int64 { 1 }
let x = foo() // infers Int, returns `0`
let y: Int64 = foo() // explicitly Int64, returns `1`The least breaking solution for the consumer would be to keep If there is a solution that can create the least disruption for the vast amount of consumers who are all working exclusively on 64-bit platforms that would be ideal. It doesn't make a ton of sense making disruptive changes to serve the needs of a tiny fraction of the consumer base. |
|
I tested But testing it turned up something that changes the shape of the decision, so rather than reply I went and measured both options.
|
| consistency refactor (currently on this PR) | minimal fix | |
|---|---|---|
| public API changes | frames, subframes, samples, Fraction, Stride |
none |
| test files changed | 3 | 0 |
| native suite | 506 pass | 504 pass |
| wasm32 | not yet run | 554 tests / 128 suites pass |
| fixes the 32-bit trap | yes | yes |
Branch: mansbernhardt:experiment/minimal.
The clamp is safe for the same reason the first version of this PR was: these values are only ever used as an upper bound, and a subframe count that large is itself unrepresentable in a 32-bit Int, so clamping still bounds the entire representable domain. On 64-bit it never engages.
Suggestion
Land the minimal fix to close the 32-bit trap with zero disruption to the 64-bit majority, and treat API consistency as its own deliberate change later — because consistency now unavoidably means choosing types for properties that can't be overloaded, which is an API decision rather than a mechanical refactor, and it deserves to be made on its own terms rather than as a side effect of a bug fix.
That said, this is your library and you've already said you'd like the consistency work here. The full refactor is pushed and green if you'd prefer it — just say which and I'll set the PR to match. I'd rather give you the measurements than argue for one.
|
Thanks for doing the exploratory work on this.
If the properties were converted to functions (
The simplicity of this approach without API changes makes the most sense at this point in time. My only hesitation is having values silently clamp instead of returning actual true values, if that behavior is not obvious at the callsite for consumers. As just one example, audio samples @ 48KHz overflows
Big-picture, yes - you're right. If we adopt There is one other possibility I might entertain at this junction before we ratify a solution. It wouldn't be entirely out of form to conditionally substitute |
|
Built and measured your fence idea rather than replying — it works, and it answers your clamping objection cleanly. Three options, all green
Branch: The shape is a single alias rather than fences at each signature, which keeps the duplication down: #if _pointerBitWidth(_64)
public typealias TimecodeTotalCount = Int
#elseif _pointerBitWidth(_32)
public typealias TimecodeTotalCount = Int64
#else
#error("Unsupported pointer width — TimecodeTotalCount needs a mapping for this platform.")
#endifApplied to total subframe counts and total sample counts — the two domains that provably overflow — plus the internal arithmetic between them. On a 64-bit build the alias is One cost worth knowing before you pick itInternal code has to be written alias-aware, and a 64-bit build will not catch mistakes. Because the alias is That's an ongoing maintenance tax rather than a one-off, and it makes the WASM CI job in #89 load-bearing rather than nice-to-have — without a 32-bit build in CI, this class of breakage lands silently. One thing I could not carry
Happy to set this PR to whichever of the three you prefer — say the word and it's one push. If it were mine I'd take the fence: it fixes the trap, gives true values everywhere, and costs 64-bit consumers nothing. But the maintenance tax above is real and you're the one who'll carry it. |
|
I'd probably move toward something more generic for an alias name like It's possible to swap in the alias for However, the more I dig into this the more it becomes evident there is no trivial way to do it cleanly without some form of compromise for the consumer. I'm increasingly leaning toward a new major version release where the entire codebase would adopt specific bitwidth types for public API consistently at every overflow pinch point, measured not just against current upper bounds but taking into consideration the potential for larger frame rates in future. |
|
On the pin, since it's the one practical thing outstanding on our side: we currently track this fork branch by Would you consider landing either the clamping or the fence variant as a 3.1.x patch in the meantime? Both are zero-public-API-change on 64-bit and green (504 native, 554 on wasm32), so neither pre-empts nor constrains the major version you're planning — they'd just close the 32-bit trap for anyone hitting it today, and let us move back onto a released tag. Entirely your call, and no urgency from our side; the pin is stable. Happy to wait for the major version if you'd rather do it once, properly. On the alias name — agreed that |
|
I would be amenable to the alias solution as a stop-gap for version 3. If we commit to that, is it feasible to widen Keep in mind that, as you may have noticed, a number of unit tests originally were simply fenced off from running on 32-bit architectures at all where some 64-bit methods were evaluated. So the tests being green on a WASM test run doesn't necessarily give a full picture of things that could still trap on a 32-bit system. It would be worth checking any areas of the tests that are exempting armv7 or i386 or similar targets and see if they can be refactored or even un-fenced now that we are supporting 64-bit integer widths where necessary in the codebase. |
…tion
Stop-gap for version 3, per review: a platform-conditional alias rather than
widening the public API for everyone.
#if _pointerBitWidth(_64)
public typealias PlatformInt = Int
#elseif _pointerBitWidth(_32)
public typealias PlatformInt = Int64
#endif
On 64-bit the alias IS Int, so the public API is byte-identical and consumers
see no change at all — 504 tests pass with zero test-file changes. On 32-bit
(wasm32, watchOS armv7k/arm64_32) it becomes Int64 at the points that provably
overflow there:
- total subframe counts — at `.max100Days` this is >= 16_588_800_000 for every
frame rate at the 80/100 subframe bases
- total audio sample counts — 4_147_200_000 at 24 hours / 48 kHz, i.e. the
library's ORDINARY limits rather than a hypothetical future frame rate
- `Fraction`'s numerator and denominator, plus its internal arithmetic
Widening `Fraction` removes the last narrowing the earlier attempts had to
leave in place: `Timecode.rationalValue` previously squeezed a 64-bit subframe
count back into an `Int`, so a timecode beyond ~Int32.max subframes had no
representable rational value on a 32-bit platform. That conversion and its
apology comment are both gone.
Left as `Int` deliberately, because they are components rather than pinch
points: `Components`' h/m/s/f, `FrameCount.subFrames`, and `FeetAndFrames`.
Total FRAME counts also remain `Int`. They fit today — 120 fps over 100 days is
1_036_800_000, 30 bits — but that is the headroom noted in review: 240 fps is 31
bits and 480 fps overflows. They are a pinch point for a future rate, not a
current one, and belong with the major-version pass rather than a stop-gap.
|
Yes — feasible, and done. Branch
Widening it also removed the last narrowing the earlier attempts had to leave in: Verified:
On "the only remaining outlier" — not quite, and the exception is yoursWith
So frames are a pinch point for a future rate rather than a current one. I deliberately left them out of the stop-gap: widening them reaches Happy to point this PR at that branch whenever you want it, or leave it as a branch to look at first. |
Excellent.
That's an acceptable compromise for version 3 and will likely carry us into the next few years before we start to see consumer and professional software/hardware see higher frame rates as commonplace. At which time a major version bump with other package-wide refactors may be in order. I've had a few ideas for refining or reworking more general type ergonomics for a future version 4, which could also carry the larger bit widths making them explicit for 32-bit platforms.
Whatever works - if you want to revert or force push on this PR then we can review and tweak before merging. Thanks so much for your work here. I appreciate the viability woodshedding and the detailed explainers of the progress. |
Follows the review point that some tests were fenced off from 32-bit entirely,
so a green WASM run does not prove much on its own. That was right, and the one
fence in the suite was the worst possible one to have:
// these integers result in overflow on armv7/i386 (32-bit arch)
#if !(arch(arm) || arch(i386))
#expect(frameRate.maxTotalSubFrames(in: .max100Days, base: .max80SubFrames)
== 2_592_000 * 100 * 80)
That is the assertion for the exact bound that traps on 32-bit — fenced off from
the only platforms that had the bug. Typing the expected value to `PlatformInt`
lets it run everywhere; on 64-bit the arithmetic is unchanged. `Tests/` now has
zero architecture fences.
Also adds an `Int` companion to `.samples(_:sampleRate:)`, which the alias turns
out to REQUIRE on 32-bit: with only `Int64` and `Double` overloads visible, an
ordinary literal expression like `.samples(48000 * 2, sampleRate: 48000)` is
ambiguous, because `Int` is Swift's default integer-literal type and matches
neither exactly. Without it the alias would silently stop ordinary call sites
compiling on exactly the platforms it exists to serve.
The companion itself must be fenced to `_pointerBitWidth(_32)`: on 64-bit
`PlatformInt` IS `Int`, so an unconditional companion is an `invalid
redeclaration`. Any overload added alongside an aliased one carries the same
constraint — worth knowing before this pattern spreads.
Consequence, stated plainly: on 32-bit, a large literal now resolves to the
`Int` companion and needs explicit typing (`4_147_200_000 as PlatformInt`). That
is a compile error rather than a silent truncation, and the sample tests are
typed accordingly.
Native: 504 tests in 55 suites.
28a73c4 to
2f020dc
Compare
|
Done — this PR now points at the stop-gap: Your point about the fences was right, and worth more than I expected. There was exactly one architecture fence in the whole suite, and it was the worst one to have: // these integers result in overflow on armv7/i386 (32-bit arch)
#if !(arch(arm) || arch(i386))
#expect(frameRate.maxTotalSubFrames(in: .max100Days, base: .max80SubFrames)
== 2_592_000 * 100 * 80)That is the assertion for the exact bound this PR fixes, fenced off from the only platforms that had the bug. Typing the expected value to Two things the alias turns out to require1. An But the companion cannot be unconditional: on 64-bit 2. Large literals on 32-bit now need explicit typing — Correcting my own claim, since you predicted this exactlyMy "565 tests pass on wasm32" figure was from my project's test lane running against the pinned library — it never ran your suite. Your suite still cannot build for wasm32, for two reasons that are pre-existing and unrelated to this PR, both verified against a pristine clone of
After this PR, the only wasm32 test-build errors remaining are those two classes. Nothing new is attributable to this change, and the sample-test errors that were attributable are gone. So you were right that a green WASM run does not give the full picture — in this case it was not even measuring what I implied it was. Thanks for the nudge; it turned up the one fence that mattered. |
Int overflow on 32-bit platforms (wasm32, watchOS)Int overflow on 32-bit platforms (wasm32, armv7)
Great - that's the ideal scenario. The fences were just workarounds originally. As I mentioned, the early scope of this package was targeting 64-bit Apple platforms which has since expanded to Linux, Android, and now WASM which is 32-bit. The fences in the tests were just kludges.
It's possible this may be trivially solved unless there conflicts that present themselves. func foo(value: PlatformInt) { print("Int", value) }
@_disfavoredOverload
func foo(value: Double) { print("Double", value) }foo(value: 1) // "Int 1"
foo(value: 2.0) // "Double 2.0"
Again, this may be largely ameliorated with overload precedence as noted above. foo(value: 0x7FFFFFFFFFFFFFFF) // Works (Int64.max)
foo(value: 0x8000000000000000) // Error: overflow (Int64.max + 1)We do not want to specifically provide |
|
The PR tests are all green except watchOS which needs a little bit of fixup. |
Replaces the fenced `Int` companion with `@_disfavoredOverload` on the `Double` overload, per review. Strictly better on every axis: - an integer literal resolves to `PlatformInt` instead of being ambiguous - a full 64-bit literal compiles UNANNOTATED on 32-bit — the explicit `4_147_200_000 as PlatformInt` typing the companion forced is reverted - no `#if _pointerBitWidth(_32)` fence around an overload, so the pattern does not have to spread - and the reason that matters most: the companion silently narrowed to `Int`, which hid from the consumer that a 64-bit value was required at all. Overload precedence surfaces the requirement instead of papering over it. Verified: 504 native tests; on wasm32 the test target builds with ZERO errors beyond the two pre-existing classes unrelated to this PR (the `@Test`/`@Suite` `@section`/`@const` macro errors, and the oversized numeric-string literals in the string-parsing and FeetAndFrames tests, both present on a pristine `main`).
|
That works, completely — and it is better than what I had on every axis. Tested rather than assumed:
Pushed. The explicit Your reasoning about why is the part I had backwards. I was treating the ambiguity as the problem to make disappear, and an State now: 504 native tests, and on wasm32 the test target builds with zero errors beyond the two pre-existing classes I described (both verified against a pristine clone of Nothing outstanding from my side — yours to review and tweak. |
|
Thanks - I can tail this PR with some tweaks now before merging. |
|
It may be a bit of over-engineering, but I broadened the Caught a few Updated docs and made sure docs are building without issues. All tests are green now (with the caveat that Android and WASM are build-only jobs until #89 is resolved and unit tests are added for WASM). I will merge this down to main now. |
The bug
TimecodeFrameRate.maxTotalSubFrames(in:base:)computes its product directly inInt:With
extent == .max100Daysthat product exceedsInt32.maxfor every frame rate. The smallest case, 23.976 fps at 80 subframes, is alreadyso the multiplication traps on overflow on any 32-bit platform — wasm32, and watchOS
armv7k/arm64_32.Because the bound is recomputed inside every wrapping add (
sfcNew.clamped(to: 0 ... maxSubFrameCountExpressible)), this makes all arithmetic on a.max100Daystimecode trap on those platforms, no matter how small the operands are.Repro (wasm32)
Observed in a browser, wasm32 debug build:
Construction, comparison,
max(by:)and.realTimeValueall work; only arithmetic under.max100Daystraps.Worth noting this is easy to hit without ever choosing
.max100Daysdeliberately: our wrapper type sets it on everyTimecodeit constructs, so every timecode operation trapped once we started building for wasm32.The fix
Compute in
Int64, saturate on return:Int64.max, so the clamp never engages. The existing exact-value assertions inTimecodeFrameRate_Properties_Tests.properties()still hold.clamped(to:)range, or a>comparison against asubFrameCount. AsubFrameCountthat large is itself unrepresentable in a 32-bitInt, so saturating atInt.maxstill bounds the entire representable domain.Tests
Two regression tests in
TimecodeFrameRate Properties Tests.swift:maxTotalSubFramesDoesNotOverflowOn32Bit()— every frame rate × every subframe base at.max100Days; asserts the exact product on 64-bit andInt.maxon 32-bit, and thatmaxSubFrameCountExpressiblestays consistent.max100DaysArithmeticDoesNotTrap()— the wrapping add above.Full suite green locally: 506 tests in 55 suites passed.
One suggestion, happy to do it separately
The wasm CI jobs added in #87 (mine) run
swift buildonly. This defect compiles perfectly and traps at runtime, so a build-only job structurally cannot catch it — and the tests above would have, had the suite run under wasm32. If you'd like, I can follow up with a PR that runsswift teston the wasm jobs via wasmtime or Node.